Add kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5 test scripts and results by ZTWHHH · Pull Request #148 · mm-evaluation/simple-mmeval

ZTWHHH · 2026-06-10T15:26:01Z

Description

Test scripts and result.json artifacts for the 6 new model series added in the companion main-branch PR #147. All tests run with --max_new_tokens 2048. Layout follows PR #118.

This PR contains test scripts/results only.

scripts/<series>/<series>-<modality>.sh
work_dirs/<series>/<model>-<modality>/result.json

Series, variants, and modalities tested (28 result.json files; all PASS):

Series	Tested variants	Modalities
`kimi-vl`	Kimi-VL-A3B-{Instruct,Thinking}	text, multi-image-interleave
`minicpm-v-4d5`	MiniCPM-V-4_5	text, multi-image-interleave
`glm-4d1v`	GLM-4.1V-9B-{Thinking,Base}	text, multi-image-interleave
`ovis2`	Ovis2-{1B,2B,4B,8B}	text, multi-image-interleave
`aria`	Aria	text, multi-image-interleave
`qwen3d5`	Qwen3.5-{0.8B,2B,4B,9B}	text, multi-image-video-interleave

Type of Change

Testing

Tested locally
Added example script (if new model/dataset)

Test command used:

bash scripts/<series>/<series>-<modality>.sh

Checklist

Updated mmeval/registery.py (if new model)
Updated README (if new model/dataset)
Code follows existing style

Test scripts and result.json files for the model series introduced in the companion main-branch PR: - kimi-vl - minicpm-v-4d5 - glm-4d1v - ovis2 - aria Each series has two scripts (text, single-image-start) and corresponding work_dirs/<series>/<model>-<modality>/result.json under the same naming convention used elsewhere in this branch. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

Test scripts and result.json files for the Qwen3.5 multimodal series, to complement the matching main-branch entry. Layout follows PR mm-evaluation#118. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

…iant Per official capability: - kimi_vl: text + multi-image-start. Both Kimi-VL-A3B-{Instruct,Thinking}. - minicpm_v_4d5: text + multi-image-start. MiniCPM-V-4_5. - glm_4d1v: text + single-video-start. GLM-4.1V-9B-{Thinking,Base}. - ovis2: text + multi-image-start. Ovis2-{1B,2B,4B,8B}. - aria: text + multi-image-start. Aria. - qwen3d5: text + multi-image-video-interleave. Qwen3.5-{0.8B,2B,4B,9B}. Each variant has both a text result.json and a broader-modality result.json under work_dirs/<series>/<model>-<modality>/. Single-image tests were dropped in favor of multi-image-start (which fully covers single-image).

Per official capability + integration analysis: - kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria: multi-image-interleave (image positions anywhere in the prompt; matches official chat-template / model.chat / processor input handling) - qwen3d5: multi-image-video-interleave (unified vision-language; native Qwen3VLProcessor handles interleaved image+video+text) multi-image-start covers single-image and was redundant; dropped. minicpm_v_4d5 inference file: parse_input was concatenating all text at the end and breaking interleave order; fixed to append text chunks in position.

Bumped --max_new_tokens from 512 to 2048 in all 12 new test scripts. Re-ran every variant on text + the broadest correct modality: - kimi_vl: text + multi-image-interleave (both Instruct & Thinking) - minicpm_v_4d5: text + multi-image-interleave - glm_4d1v: text + multi-image-interleave (both Thinking & Base) - ovis2: text + multi-image-interleave (1B, 2B, 4B, 8B) - aria: text + multi-image-interleave - qwen3d5: text + multi-image-video-interleave (0.8B, 2B, 4B, 9B) Reviewed every result.json: 0 empty responses, 0 truncated-at-noise, correct media-count match per row, correct chat-template formatting. The single Qwen3.5-9B sample 9 hit the 2048 budget (~8.4 KB output) which is the intended behaviour at max_new_tokens=2048.

Copilot

Pull request overview

Adds runnable test scripts and corresponding result.json artifacts for the 6 newly integrated model series (companion to PR #147), using --max_new_tokens 2048 and the established scripts/<series>/... + work_dirs/<series>/... layout.

Changes:

Add shell scripts to run local JSON sample evaluations for each new series/modality.
Add work_dirs/**/result.json artifacts capturing PASS runs for the listed variants/modalities.
Cover both text-only and multimodal (multi-image-interleave / multi-image-video-interleave) scenarios depending on series.

Reviewed changes

Copilot reviewed 40 out of 40 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
scripts/qwen3d5/qwen3d5-text.sh	Runs text-only sample evals for Qwen3.5 variants (0.8B/2B/4B/9B).
scripts/qwen3d5/qwen3d5-multi-image-video-interleave.sh	Runs multi-image-video-interleave sample evals for Qwen3.5 variants.
scripts/ovis2/ovis2-text.sh	Runs text-only sample evals for Ovis2 variants (1B/2B/4B/8B).
scripts/ovis2/ovis2-multi-image-interleave.sh	Runs multi-image-interleave sample evals for Ovis2 variants.
scripts/minicpm-v-4d5/minicpm-v-4d5-text.sh	Runs text-only sample eval for MiniCPM-V-4_5.
scripts/minicpm-v-4d5/minicpm-v-4d5-multi-image-interleave.sh	Runs multi-image-interleave sample eval for MiniCPM-V-4_5.
scripts/kimi-vl/kimi-vl-text.sh	Runs text-only sample evals for Kimi-VL-A3B Instruct/Thinking.
scripts/kimi-vl/kimi-vl-multi-image-interleave.sh	Runs multi-image-interleave sample evals for Kimi-VL-A3B Instruct/Thinking.
scripts/glm-4d1v/glm-4d1v-text.sh	Runs text-only sample evals for GLM-4.1V Base/Thinking.
scripts/glm-4d1v/glm-4d1v-multi-image-interleave.sh	Runs multi-image-interleave sample evals for GLM-4.1V Base/Thinking.
scripts/aria/aria-text.sh	Runs text-only sample eval for Aria.
scripts/aria/aria-multi-image-interleave.sh	Runs multi-image-interleave sample eval for Aria.
work_dirs/qwen3d5/Qwen3.5-0.8B-text/result.json	Captured PASS output for Qwen3.5-0.8B text-only sample.
work_dirs/qwen3d5/Qwen3.5-2B-text/result.json	Captured PASS output for Qwen3.5-2B text-only sample.
work_dirs/qwen3d5/Qwen3.5-4B-text/result.json	Captured PASS output for Qwen3.5-4B text-only sample.
work_dirs/qwen3d5/Qwen3.5-9B-text/result.json	Captured PASS output for Qwen3.5-9B text-only sample.
work_dirs/qwen3d5/Qwen3.5-0.8B-multi-image-video-interleave/result.json	Captured PASS output for Qwen3.5-0.8B multi-image-video-interleave sample.
work_dirs/qwen3d5/Qwen3.5-2B-multi-image-video-interleave/result.json	Captured PASS output for Qwen3.5-2B multi-image-video-interleave sample.
work_dirs/qwen3d5/Qwen3.5-4B-multi-image-video-interleave/result.json	Captured PASS output for Qwen3.5-4B multi-image-video-interleave sample.
work_dirs/qwen3d5/Qwen3.5-9B-multi-image-video-interleave/result.json	Captured PASS output for Qwen3.5-9B multi-image-video-interleave sample.
work_dirs/ovis2/Ovis2-1B-text/result.json	Captured PASS output for Ovis2-1B text-only sample.
work_dirs/ovis2/Ovis2-1B-multi-image-interleave/result.json	Captured PASS output for Ovis2-1B multi-image-interleave sample.
work_dirs/ovis2/Ovis2-2B-text/result.json	Captured PASS output for Ovis2-2B text-only sample.
work_dirs/ovis2/Ovis2-2B-multi-image-interleave/result.json	Captured PASS output for Ovis2-2B multi-image-interleave sample.
work_dirs/ovis2/Ovis2-4B-text/result.json	Captured PASS output for Ovis2-4B text-only sample.
work_dirs/ovis2/Ovis2-4B-multi-image-interleave/result.json	Captured PASS output for Ovis2-4B multi-image-interleave sample.
work_dirs/ovis2/Ovis2-8B-text/result.json	Captured PASS output for Ovis2-8B text-only sample.
work_dirs/ovis2/Ovis2-8B-multi-image-interleave/result.json	Captured PASS output for Ovis2-8B multi-image-interleave sample.
work_dirs/minicpm-v-4d5/MiniCPM-V-4_5-text/result.json	Captured PASS output for MiniCPM-V-4_5 text-only sample.
work_dirs/minicpm-v-4d5/MiniCPM-V-4_5-multi-image-interleave/result.json	Captured PASS output for MiniCPM-V-4_5 multi-image-interleave sample.
work_dirs/kimi-vl/Kimi-VL-A3B-Instruct-text/result.json	Captured PASS output for Kimi-VL-A3B-Instruct text-only sample.
work_dirs/kimi-vl/Kimi-VL-A3B-Instruct-multi-image-interleave/result.json	Captured PASS output for Kimi-VL-A3B-Instruct multi-image-interleave sample.
work_dirs/kimi-vl/Kimi-VL-A3B-Thinking-text/result.json	Captured PASS output for Kimi-VL-A3B-Thinking text-only sample.
work_dirs/kimi-vl/Kimi-VL-A3B-Thinking-multi-image-interleave/result.json	Captured PASS output for Kimi-VL-A3B-Thinking multi-image-interleave sample.
work_dirs/glm-4d1v/GLM-4.1V-9B-Thinking-text/result.json	Captured PASS output for GLM-4.1V-9B-Thinking text-only sample.
work_dirs/glm-4d1v/GLM-4.1V-9B-Thinking-multi-image-interleave/result.json	Captured PASS output for GLM-4.1V-9B-Thinking multi-image-interleave sample.
work_dirs/glm-4d1v/GLM-4.1V-9B-Base-text/result.json	Captured PASS output for GLM-4.1V-9B-Base text-only sample.
work_dirs/glm-4d1v/GLM-4.1V-9B-Base-multi-image-interleave/result.json	Captured PASS output for GLM-4.1V-9B-Base multi-image-interleave sample.
work_dirs/aria/Aria-text/result.json	Captured PASS output for Aria text-only sample.
work_dirs/aria/Aria-multi-image-interleave/result.json	Captured PASS output for Aria multi-image-interleave sample.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

ZTWHHH · 2026-06-17T03:31:31Z

+            {
+                "role": "assistant",
+                "response": [
+                    "Thinking Process:\n\n1.  **Analyze the Request:** The user is asking a simple arithmetic question: \"What is the result of 1 plus 1?\"\n\n2.  **Identify the Core Operation:** The operation is addition. Specifically, $1 + 1$.\n\n3.  **Perform the Calculation:** $1 + 1 = 2$.\n\n4.  **Formulate the Answer:** The answer should be direct and clear. \"2\" or \"The result is 2.\"\n\n5.  **Review for Constraints:** The prompt is straightforward, no special constraints or context provided.\n\n6.  **Final Output Generation:** State the result clearly.cw\n</think>\n\nThe result of 1 plus 1 is **2**.\n"


Thanks for the review. I reverted the Qwen3.5 cw cleanup because it appears to be part of the model's generated behavior rather than a purely mechanical boundary token. To preserve official behavior and evaluation comparability, the raw artifact is kept in the saved output. </think> is kept because it is useful for separating the reasoning section from the final answer.

ZTWHHH · 2026-06-17T03:31:56Z

+            {
+                "role": "assistant",
+                "response": [
+                    "Thinking Process:\n\n1.  **Analyze the Request:** The user is asking a very basic arithmetic question: \"What is the result of 1 plus 1?\"\n\n2.  **Identify the Core Operation:** The operation is addition ($1 + 1$).\n\n3.  **Calculate the Result:** $1 + 1 = 2$.\n\n4.  **Formulate the Answer:** The answer should be direct and clear. \"2\" or \"The result is 2.\"\n\n5.  **Review for Constraints/Context:** There are no special constraints (e.g., \"in binary,\" \"in a specific context\"). The standard base-10 arithmetic is implied.\n\n6.  **Final Output:** \"2\" or \"The result is 2.\" I'll go with a simple, direct sentence.\n\n7.  **Final Check:** Does this answer the user's question accurately? Yes.cw\n</think>\n\nThe result of 1 plus 1 is **2**.\n"


Same as the 4B case: reverted the cw cleanup so the raw artifact is preserved as model-generated behavior, and kept </think> as the reasoning/answer boundary.

ZTWHHH · 2026-06-17T03:32:15Z

+            },
+            {
+                "role": "assistant",
+                "response": "The result of 1 plus 1 is 2.<|user|>"


ZTWHHH · 2026-06-17T03:32:27Z

+            },
+            {
+                "role": "assistant",
+                "response": "The capital city of China is Beijing.<|user|>"


Same fix as the previous comment — trailing <|user|> is stripped from this row as well; the answer content is unchanged.

ZTWHHH · 2026-06-17T03:32:40Z

+            },
+            {
+                "role": "assistant",
+                "response": "The result of adding 1 plus 1 is 2.<|im_end|>"


I added a conservative strip for a single trailing <|im_end|> in aria and updated the affected result.json. The answer content is unchanged.

ZTWHHH · 2026-06-17T03:32:57Z

+            },
+            {
+                "role": "assistant",
+                "response": "The capital city of China is Beijing.<|im_end|>"


Same fix as the previous comment — trailing <|im_end|> is stripped from this row as well; the answer content is unchanged.

ZTWHHH · 2026-06-17T03:33:08Z

+            },
+            {
+                "role": "assistant",
+                "response": "<think>The image displays a pickup truck in motion on a dirt road. The truck is a GMC model, as indicated by the prominent GMC logo on the front grille. The vehicle is painted in a metallic gray color. It features a crew cab design with four doors. The front grille is large and black with a honeycomb pattern. The headlights are sleek and modern, with LED daytime running lights. The truck has large, black, off-road tires with visible tread patterns. The suspension appears to be lifted, giving the truck a high ground clearance. The truck is kicking up a significant amount of dust from the road, suggesting it is moving at a moderate speed. In the background, there are reddish-brown rock formations, indicating a desert or canyon-like environment. The lighting suggests it is daytime, with sunlight illuminating the truck and casting shadows. The overall impression is of a rugged, capable, and stylish off-road vehicle.</think><answer>The image shows a GMC pickup truck, likely a model from the 2020s, in motion on a dirt road. The truck is painted in a metallic gray color and features a crew cab design with four doors. It has a prominent black front grille with the GMC logo in the center. The headlights are modern and sleek, with LED daytime running lights. The truck is equipped with large, black, off-road tires that are kicking up a significant amount of dust, indicating it is moving. The suspension is lifted, providing high ground clearance. The background consists of reddish-brown rock formations, suggesting a desert or canyon environment. The lighting is bright, indicating daytime.\n\n<|begin_of_box|>A metallic gray GMC pickup truck, likely a crew cab model, is shown in motion on a dirt road, kicking up dust. It features a black front grille with the GMC logo, modern headlights, and large black off-road tires. The background shows reddish-brown rock formations, suggesting a desert or canyon setting.<|end_of_box|></answer><|user|>"


ZTWHHH · 2026-06-17T03:33:21Z

+            },
+            {
+                "role": "assistant",
+                "response": "The image showcases a GMC truck in action, captured in an outdoor setting. The truck is prominently displayed in a dynamic pose, suggesting it is moving at a high speed. It's a four-door pickup truck with a robust, muscular design that emphasizes its off-road capabilities. \n\nThe front of the truck features a distinctive GMC grille with the logo centered, flanked by sleek, angular headlights. The truck has a modern, aerodynamic body with a dark metallic finish. The wheels are black and robust, adding to the rugged appearance. Dust is being kicked up from the rear tires, indicating that the truck is driving on an unpaved, rocky terrain. The background consists of arid, desert-like cliffs, enhancing the feeling of adventure and durability. This scene portrays the truck's power, reliability, and readiness for challenging environments.<|im_end|>"


Cleaned every row in this file with the same trailing <|im_end|> strip. The answer content is unchanged.

Per AI-review on PR mm-evaluation#148, strip trailing/inline chat-template boundary tokens from result.json files where they appeared as known mechanical artifacts: - aria: trailing <|im_end|> (chat-template end-of-turn) - glm_4d1v base: trailing <|user|> (start-of-next-turn artifact) - qwen3d5: mechanical 'cw' before </think> (kept </think> as it separates reasoning from the final answer) Semantic structure such as <think>...</think>, <answer>...</answer>, and <|begin_of_box|>...<|end_of_box|> is preserved.

ZTWHHH and others added 2 commits June 10, 2026 15:24

Add qwen3d5 test scripts and results

1d0182c

Test scripts and result.json files for the Qwen3.5 multimodal series, to complement the matching main-branch entry. Layout follows PR mm-evaluation#118. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>

ZTWHHH changed the title ~~Add test scripts and results for 5 new model series~~ Add kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5 test scripts and results Jun 10, 2026

ZTWHHH mentioned this pull request Jun 11, 2026

Add kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5 infer code #147

Merged

10 tasks

ZTWHHH added 2 commits June 11, 2026 08:10

ZTWHHH requested a review from Irisicy4 June 14, 2026 15:12

Irisicy4 requested a review from Copilot June 14, 2026 15:15

Copilot started reviewing on behalf of Irisicy4 June 14, 2026 15:15 View session

Copilot AI reviewed Jun 14, 2026

View reviewed changes

ZTWHHH force-pushed the add-five-new-model-series-test-result branch from 8f203c5 to 52d708d Compare June 17, 2026 03:19

ZTWHHH merged commit 756bb5f into mm-evaluation:test-result Jun 17, 2026

Navigation Menu

Uh oh!

Add kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5 test scripts and results - #148

Add kimi_vl, minicpm_v_4d5, glm_4d1v, ovis2, aria, qwen3d5 test scripts and results#148
ZTWHHH merged 6 commits into
mm-evaluation:test-resultfrom
ZTWHHH:add-five-new-model-series-test-result

ZTWHHH commented Jun 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

ZTWHHH Jun 17, 2026

Uh oh!

ZTWHHH Jun 17, 2026

Uh oh!

ZTWHHH Jun 17, 2026

Uh oh!

ZTWHHH Jun 17, 2026

Uh oh!

ZTWHHH Jun 17, 2026

Uh oh!

ZTWHHH Jun 17, 2026

Uh oh!

ZTWHHH Jun 17, 2026

Uh oh!

ZTWHHH Jun 17, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

ZTWHHH commented Jun 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of Change

Testing

Test command used:

Checklist

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ZTWHHH commented Jun 10, 2026 •

edited

Loading